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ABSTRACT 


An approach to simultaneous interpretation of objects in complex 
structures so as to maximize a combined utility function is presented. 
Results of the application of a computer software system to assign meaning 
to regions in a segmented image based on the principles described in this 
paper and on a special interactive sequential classification learning system, 
which is referenced, are demonstrated. 
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Introduction 


This paper presents principles underlying a soft- 
ware system which interprets relatively complex 
structures composed of many components that are 
mutually dependent in their meaning and properties. 
These principles are, in a sense, a cross-breed 
between Artificial Intelligence and pattern recognition. 

Most of the theoretical works on pattern recogni- 
tion address themselves to the problem of assigning a 
label (meaning) to a given object so as to maximize 
a utility function. The utility function is defined 
using: (1) the a priori probability of an object to be of 

a given label, (2) various measured features of the 
object, and (3) the probability that an object of a given 
label will have those features. 

The research in pattern recognition was mainly 
aimed at development of: (1) techniques for learn- 
ing various probabilities of events in the specific 
problem domain (if not known a priori), (2) ways of 
making efficient use of features to distinguish between 
objects of different classes, and (3) various tech- 
niques used to assign a meaning to an object so as to 
maximize the utility functions. However, structures 
in the real world have a tendency to be composed of 
many components, each of which is an autonomous 
object, and the mutual structure is usually very help- 
ful in recognizing them. This is true for speech and 
written texts (meaningful sentences), as well as for 
general pictorial information (like various images 
used in biomedical diagnosis or product quality 
control) . 

From all approaches to pattern recognition, the 
closest approach to ours is the probabilistic gram- 
mars (Ref. 1). Both try to parse complex structures, 
but the mechanism (syntax) which represents knowl- 
edge (semantic) is entirely different. The parsing 
(meaning assignment) algorithm suggested in this 
paper, even though it may be extended to handle the 
parsing of certain grammars, was designed mainly 
to parse structures using real-world knowledge 
represented using sequential classification 
mechanisms . 

Many researchers in Artificial Intelligence (Al) 
address themselves to the specific problems arising 
when attempts are made to design machines which 
interact intelligently with relatively unconstrained 
environments. Whenever it was necessary to sense 
an environment in some way, a real-world pattern 


recognition problem arose. However, the techniques 
applied were usually of a computer science nature, 
such as: (1) algorithms for searches through trees of 

states, (2) integer programming and constraint satis- 
fying, (3) logic, (4) procedural knowledge, (5) data 
bases, (6) special languages, (7) interactive pro- 
gramming, and other such software products. These 
approaches were sufficiently powerful for some prob- 
lems, but the absence of probabilistic modeling and 
the lack of well-defined utility functions resulted in 
an ad hoc system. There was no effective way to 
evaluate various decision processes and performance, 
except by applying the resulting system to various 
examples and evaluating the output manually. A 
typical system is described in Ref. 2. 


Many of the ideas presented in this paper arose in 
the context of AI research. The author believes that 
combining AI techniques with statistical pattern 
recognition will result in an approach which will be 
significantly more powerful than each used alone, at 
least in the area of pictorial pattern recognition. 


Problem Domain 

The problem domain to which the specific software 
system, whose principles are described in the paper, 
is applied is pictorial pattern recognition. This type 
of application differs from character recognition in 
that the objects to be recognized are not line- shaped, 
but are areas, and are not purely black objects on a 
white background; they may vary in their light inten- 
sity. Examples of analyzing left ventricular angio- 
grams and road scenes are shown at the end of this 
paper (Figs. 1 - 3). Using various clustering tech- 
niques and edge detectors, the raw picture is reduced 
to a few connected domains (regions) of almost uni- 
form intensity. The reduction is done with conserva- 
tive enough parameters so that the images of two 
distinct objects will not be merged into one domain. 

As a result, the image of a single object is likely to 
be broken into a few domains. This clustering algor- 
ithm is described in Ref. 3. Experiments proved that 
there is no way to get a perfect segmentation (having 
one segment per imaged object) using only measure- 
ments at grid points without using semantics of the 
expected scene. Shadows and changes in light 
reflectance as a result of surface direction change and 
local textures on the surface of a single object are 
sometimes more distinguishable then the real bound- 
aries of the object, and as such prohibit a perfect 
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segmentation without understanding the image. 

Hence, in relatively early phases with quite a few 
redundant segments, interpretation of the image is 
needed. 

Suppose that there is a segmented image with 
R(i), i = 1, ■ • « , N domains (regions, segments), 
and let B(i, j) stand for the common boundary between 
domain R(i) and domain R(j) if it exists, {B{i, j) con- 
sists of one or more continuous paths). A global 
interpretation of the scene will mean an assignment 
of meanings to each domain. Let INT stand for a 
global interpretation function, then 


INT: R{i) 


Model A; ’’First Order 11 Structure 

As was mentioned above, what is required of a 
practical system to interpret involved structures of 
objects is a way of reducing the number of relations 
involved. This calls for making special simplifying 
assumptions on the structure of the dependence 
between objects in the definition of the utility function. 
One such assumption that seems to be a good com- 
promise between the complexity of expression (1) and 
the oversimplification of expression (2) is to consider, 
in addition to the individual regions, pairs of adjacent 
regions, i.e., pairs of regions that have common 
boundaries. The new utility function which will 
approximate the global probability for a given inter- 
pretation function INT will be the following expression: 


where R(i) is an element of the set of all domains, 
and j is one of the few possible labels (interpretations) 
for domains (INT (R(i)) can be equal to INT (R(j)) 
when i * j). The range of labels is defined by the 
scope of possible objects in the expected image. The 
reader should remember that INT (R(i)) - j should 
read, M the domain R(i) is part of the image of the 
real-world object of meaning j, Tl because often the 
domain is only part of the image of the object. Our 
task is to find an interpretation function INT such that 
the probability 


^ = j INT(R(i)) | measurements inside R(i) } 

xl"l P{B{i,j) is between INT(R(i)} and INT(B(i, j)) 
B(i,j) |features of R(i), R(j), and B(i, j)| 

(3) 


where B(i, j) is the boundary between R(i) and R(j). 


pJlNT(R(i)) i= 1 , N | measurements of segmented image | (1) 

which is our utility function, is maximized over all 
possible INT('). Expression (1) can be read: the 
probability of the event: ”R(i) is part of the image of 
an object of label INT(R(i)) for i = 1, • ♦ • , N, given 
all measurements of the segmented image. 11 The 
measurements are the gray level readings at the vari- 
ous points of the image and the boundary lines which 
segment the image. 

The process of looking at each individual segment 
as it stands alone and determining the object for 
which it is part of the image is the classical pattern- 
recognition problem. Assigning meaning to each 
domain independently of interpretation and features 
of other domains will be equivalent to finding an inter- 
pretation function which maximizes the following 
utility function: 

l<i<N *HlNT{R(i)) | measurements inside R(i)| (2) 


over all possible interpretation functions. However, 
by considering individual objects one at a time, as if 
they are independent, much relevant information is 
lost. There is strong mutual dependence of one object 
on the others for properties and meaning. For 
instance, the image of the sky is always above the 
imaged hill, assuming that the camera is in an upright 
position. Hence, if a hill has been determined, then 
an area strictly below it must not be sky. It may be 
a lake if it is blue or a road if it is colorless. 

The problem with using mutual dependence between 
different domains interpretations is the overwhelming 
number of relations that exist. In a set of N domains, 
there are 2^ possible relations. The next two sections 
describe two approaches which allow reduction of 
combinatorics and, hence, allow a practical use of 
the mutual dependence of interpretations of domains 
in a segmented image to get a more reliable global 
interpretation. 


In this expression, the number of product terms we 
want to maximize simultaneously is 0(N), assuming 
that for a given region the average number of regions 
that have, common boundaries with it is constant, 
independent of the total number of regions in the seg- 
mented scene. An alternative approach is to allow 
binary relations between any pairs of regions, in 
which case for a scene that is segmented into N 
domains, there always will be N + N^ terms in the 
expression that need to be maximized. The nice thing 
about Eq. (3) is that there is a practical way of maxi- 
mizing it, using integer programming tree search 
with pruning (see term definitions in Ref. 6, and 
Appendix 1 for an algorithmic description). 


Model B: Utilizing Procedural Knowledge 
Through Sequential Classification 

In the previous section the assumption is that, at 
most, binary relations between the domains will be 
sufficient and all of those relations should be con- 
sidered. This assumption constrains the possible 
features used to analyze a scene, since only a single 
region or a pair of adjacent regions may be observed 
at a time; moreover, the consideration of all binary 
relations may be redundant. A much more flexible 
mechanism will utilize sequential classification (which 
is, in a sense, procedural knowledge) to make use of 
whatever combination of features is important for 
recognition of a given object. The idea is that the 
analysis of an object will use the structure of other 
objects only when essential and can go as far in using 
many of these other domains as is convenient or bene- 
ficial. The decisions as to which features of which 
object will be helpful in finding the reaL label for an 
object with high confidence will be very specific to the 
configuration around the specific object and, hence, 
may involve properties of many other domains 
when needed. In this way, even though we allow 
involvement of many other domains in analyzing a 
domain, they will be used only as necessary and, 
hence, will not create prohibitive combinatorics. The 
mechanism of sequential classification (classification 
tree) in Refs. 3 and 4 aims at achieving just that. All 
that is required is to expand the class of features 
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which are used by the sequential classification tree to 
allow references to other objects. One such expan- 
sion is to allow, when computing P |lNT(R(i)) - j !«••} 
the use of features of region R(k) (i * k) chosen 
according to relevance to R(i). Another expansion 
will be to ask about labels (if known) of R(k) when 
computing P |lNT(R(i)) = jf (i *k). 

In this way, for instance, if a region is suspected 
to be a "hammer handle, " because it was already 
found to be elongated, and of the proper size, then the 
regions bordering the narrow far edges may be tested 
to see if they were already assigned the label “ham- 
mer head." If these are not interpreted yet, then 
check if their shape is suitable for a "hammer head, 11 
Depending on the results of these questions, the prob- 
ability of the elongated region to be labeled a "ham- 
mer handle" may increase or decrease as the result 
of the above interrogation and will lead to different 
terminal nodes of the sequential classification tree 
and, hence, different weight will be assigned to differ- 
ent labels. This type of deduction, without the explicit 
use of classification trees or probabilities, is used in 
Refs* 2, 5 and 7. Note that even though the sequen- 
tial classification mechanism allows the use of prop- 
erties and labels of many regions to analyze a region, 
it does not allow direct "feedback. 11 That is, if a 
region is not assigned meaning yet and is with proba- 
bility 0.2 "shoe" and 0. 8 "hammer head, " then this 
fact cannot be used directly to evaluate the proba- 
bility of different labels of the region suspected to be 
"hammer handle." Allowing these probabilities to 
affect each other in this level will result in a rela- 
tively complicated "feedback" equations system. 

The mutual dependence may be used as described 
in the assignment algorithm. The label assignment 
algorithm in this case (Appendix 2) will be slightly 
more complicated than in the case of first-order 
relations, but not significantly more. The added com- 
plexity results from the need to remember for each 
region which new assignments of labels to other 
regions will affect the probability of different inter- 
pretation of that region. 

The sequential assignment algorithm (Appendix 2) 
which is used to find the optimal global interpretation 
for a segmented image is built around the simple 
equality: 


PJAjO * • • OA n ) = P(A n | A r A 2 , • • * , A ft-1 ) 

* P < A n-l| A l' ■ A n-2> 

• . •P(A 2 | Aj) • P(A 1 ) 


which always holds independent of the order of 
Aj f • • *A n . The algorithm maximizes the right 
hand of the equality, where in our case for a given 
interpretation INT(*), the event A^ is "region R(I) is 
INT(R(i)), fl when R(I) is the k-th region to be assigned 
a label. All probabilities are computed for the fea- 
tures of the segmented image; the order is decided 
heuristically by the algorithm. Al is the first assign- 
ment, A2 is the second, and so on. The computation 
mechanism for the different P( - ) T s is the sequential 
classification mechanism, described briefly below. 
The major part of the assignment algorithm is to order 
the tests so as to achieve optimal INT { ■ ) rapidly. 


approximation for P(AiH ♦ • • HAJ will be 
order-dependent. That is, if 


P(R^ is a chair and is the leg of a chair) 
is approximated by 


P(R( 1) is a chair | R(2) is a leg of a chair) 
• P(R(2) is a leg of a chair) 


it may differ from 


P(R(2) is a leg of a chair j R(l) is a chair) 
♦ P(R(1) is a chair) 


which is another approximation of the combined 
probability. 

This possibility is highly undesirable and may be 
avoided by approximating the probabilities 
P(AjO ■ ■ * OA n ) by taking the geometrical averages 
of all sequential approximations. Since there are ni 
ways to order n elements, there are n! such approxi- 
mations and the problem is prohibitive combinatori- 
ally . Fortunately, in our case, in computing 
P(Aj | • • • ) only few A^’s are of any importance. 

In model A, for instance, if Aj. is the event R(I) being 
INT(R(I)), then only events relevant to A^ are A^ T s 
where R(K) is an adjacent region to R(I). Hence, if 
Ai is dependent only on 4-10 elements, then 
P(Ai | * • • ) will have no more than 24 to 2 10 differ- 
ent values depending on the order of computation. 

The weighted geometric average of the different cases 
of P(A. |.) will be 


P(A.) =(P(A. | Sl ) " 


, V W ? 

w 

* P A. S-) 2 • • 

■ p(a. I s ) n 

\ x I 2/ 

l i | nl 

o < w. y>. = i 

i 4-1 

(4) 


i 


where S j , * • • , S n are all subsets of possible events 

affecting 

Now since F(A^) is computed via the classification 
tree, Aj will affect A^ only if in some Btage questions 
about the occurrence of Aj will be asked while com- 
puting PiAJ . 


In computing the weight for the different terms 
in Eq. (4), we give equal weights to the values 
returning from the branch where Aj is assumed to 
be known before Ai and to the branch where Aj is 
assumed to become known after Ai- If this 
approach is taken, only minor modifications of 
the assignment algorithm are needed to a get 
state evaluation function where 


P{A i n. . . A n ) = P(Aj) • 


This approach has one major deficiency: since all 

the probabilities are only approximated, the resulting 


P{A 2 ) * • • • F(A n ). 
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and P(Ai) is the weighted geometrical average of the 
various order** dependent approximations to P(Ai)» 

The Sequential Classification Mechanism 
(Classification Tree) 

The classification tree is the mechanism that iB 
used to store the various probabilities and provide a 
convenient mechanism for learning those probabil- 
ities. Full description of the approach is beyond the 
scope of this paper; see Ref. 3 for a detailed descrip- 
tion of the various relevant algorithms and Ref. 4 for 
the theory. The sequential classification tree in the 
first model provides the mechanism for computing 

pjlNT(R(i)) - j | features of R(i) } 


and 

P | B(i, j) is between JNT(i) and INT(j>| features of R(i). R(j). 

In the second model, the features used to compute 
P(lNT(R(i)) = j) may include other regions 1 features 
and labels. As a result, the sequential classification 
tree provides also for locating, for a given object, the 
objects that, if they should be labeled, will require 
recomputing the probabilities of different labels of the 
given object. ThiB will happen if the sequence of 
questions about the current object R(i) leads into nodes 
of the classification tree where there are questions 
like: "Ib the label of R(j) X? " With three possible 
answers: n Yes, M M No, M and "No label assigned yet. ,r 
("No label assigned 11 means: region R(j) is not 
assumed to be known yet. ) In this case, when R(j) 
becomes known, it will affect the probabilities of the 
different possible labels of R(i), as it will lead to a 
different terminal node in the classification tree. 


Results and Conclusions 

The conclusions of this research are as follows: 

(1) There is no way to segment images into regions 
where each region will stand for the projection of 

one whole object, without the use of some model of the 
expected scene. Shadows, noise, and light changes, 
as results of surface orientation, prohibit such seg- 
mentation based only on discontinuity of color or light. 

(2) Segmentation based on discontinuity of light 
reaching the camera, when done with sensitive enough 
parameters, is very useful for data reduction as it 
generates relatively few (a few hundred) regions of 
almost uniform reflectivity structure. Each of these 
areas is part of the total image of an object, and this 
data reduction eases the shape recognition and the 
extraction of global features of the complete objects. 

(3) Only a few large areas of uniform light struc- 
ture, like the sky or the large portion of the road, 
could be recognized with confidence as they stood 
alone. For most regions, the mutual contextual con- 
straints were essential for valid recognition. This 
was especially true for small regions like shadows . 

(4) Model A, combined with assignment algor- 
ithm 1, is an effective way of allowing references to 
neighboring structure in order to recognize the smal- 
ler objects. Model B, which is more general and 
allows a more flexible deduction mechanism, is in 
implementation at the present time. 


(5) The supervised learning used for the system 
described in Ref. 3 is an effective learning system, 
but the ability to handle a large data base of manu- 
ally interpreted typical examples for automatic 
improvement of recognition is desirable. 

The results of some experiments are included at 
the end of the paper, in Figs. 1-3. These figures 
are cathode ray tube displays of digitized pictures; 

5 bits are used to display the original, and 1 bit to 
draw the boundary lines of regions superimposed on 
the original. The digitized pictures that the computer 
processes are in color, with 6 bits in each basic filter 
(18 bits total for each picture point) but only the 
5 significant bits are shown in the pictures. 

The basic (nonsemantic) segmentation, based only 
on continuity of lighting, is described in detail in 
Ref. 3. The resulting segmented image (which is 
over-segmented into small regions) is passed to the 
system, using the assignment algorithm described in 
Appendix 1, and if two adjacent regions are assigned 
the same label (meaning), they are assumed to be part 
of the same object and are merged. A few iterations 
of reducing the number of regions using the assigned 
interpretation are demonstrated with each sequence of 
experiments . 

In the outdoor scenes, the allowed objects were 
sky, hill, tree, road, car, and shadow of car. In the 
left ventricular angiogram, the allowed objects were 
the in-heart blood volume, the background organs, 
and the dark frame. The expected properties of each 
object were learned under human supervision using 
the system described in Ref. 3. 


Appendix 1 

Assignment Algorithm for Model A 

Definition of Terms 

R(I), 1=1, • • • , N is the N objects (regions, 
domains) in the segmented scene. 

B(I, K) is the common boundary of R(I) and R(K) when 
it exists. 

INTER is the set of R(I)'s which are assigned a label 
in the current state of the algorithm. 

LABEL(I) is the label currently assigned to 
R(I) € INTER. 

J = 1, ■ • * , M is the M possible labels (interpre- 
tations) that can be assigned to objects. 

P(I, J) the current score achieved by assigning 
label J to object I. 

BEST(I) is such that P(I, BEST(I)) = Max. jP(I, J)} , 
where Ii JsM, and J xb assignable to I (to allow 
backup, a label J is marked unassignable to R(I) 
if, in a given state, all resulting combinations of 
final states where I is J were tested). 

NBEST(I) is such that P(I, NBEST(I)) = MaxjJ P(I, J)}, 
where 1<J<M, J assignable to I 
J / BEST (I) 

ASSIGN (LEVEL) contains the name of region 
assigned meaning in that level. 
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Procedures 

INITfl) is the procedure which computes P(I, J), for 
all labels J, using R{1) features. This procedure 
pushes R(I) through the sequential classification 
tree for regions to get into the terminal node in 
the tree as determined by R(l) features (see Ref. 3 
for detailed description). 

SORT maintains structure allowing access to the 

object with highest confidence value (defined below) 
among the objects not interpreted yet {see Ref. 8, 
pp. 150-152). 


Algorithm Steps 

0. INTER = <t> 

1. Call INIT (I) I = 1. • ♦ * > N 

2. SEEN = 0 

LEVEL = 0 

VALUE (LEVEL) = 1 

3. Compute for I = N 


CONFIDENCE(I) = If P(I p NBEST(I)) * 0 
then p F (i! ? W B BEST(li ; else " 


11. For all objects that P(I1, Jl) was changed, 
store old values with level so that when undoing the 
recent assignment, the old values may be 
restored. 

12. Compute BEST(I), NBEST(l) for all I i INTER if 
there is an uninterpreted object II such that 
VALUE (LEVEL)* P(I1, BEST(Il)) < SEEN, then 
go to Bl. 

13 . Go to 4. 

Bl. If LEVEL = 0, then exit program. 

B2 . Restore all old values which were changed in 
current level and mark all pairs of I and J marked 
unassignable in that level as assignable. Undo the 
assignment in level, remove ASSIGN(LEVEL) 
from INTER. 

B3 . LEVEL = LEVEL - 1. 

B4 ♦ Go to 12. 


Appendix 2 

Assignment Algorithm for Model B 
Definition of Terms 

R(i) 1=1, ■ ■ \ N are the N objects (regions, domains) 
in the segmented scene. 


(if regions are allowed to be ''unidentified objects," 
then co should be replaced by special formulas). 

4. Sort R(I) by CONFIDENCE (I). 

5. Choose X 1< I-sN where R(I) has no assigned label 
yet such that 

CONFIDENCE (X) = MAX(CONFIDENCE(K)) 

K 

where: 1<K<N 

and R(K) i INTER 

6. Mark BEST (I) as unassignable for I in the cur- 
rent level and for deeper levels. (This will make 
BEST(I) not assignable again to R{I) should the 
program get back to that point). Put I in INTER. 
LABEL(I) = BEST(I). 

7. VALUE (LEVEL + 1) = VALUE( LEVEL) 

*P(I, BEST (I)) 

LEVEL = LEVEL + 1; ASSIGN (LEVEL) = 1 

8. If VALU E (LEV EL) < SEEN then go to Bl. 

9. If all objects are assigned meaning then: 

SEEN = VALU E( LEVEL). Store current global 
interpretation as best seen so far. Go to Bl. 

10, For all II that B(U1) exists, do: 

P(U, Jl) = P(I1» JD * 

P(B(I, II) is between LABEL(I) and Jl| 
features of B(I»I1) R(I) R(H)) 


INTER is the set of R(I) f s which are assigned a label 
in the current state of the algorithm. 

LABEL(I) will be the label assigned to R(I) if 
R(I) e INTER. 

J = 1, • * *, M is the M possible labels (interpretation) 
that can be assigned to objects. 

P(I, J) — * the current score achieved by assigning 
label J to object I. 

BEST (I) is such that 

P(I, BEST(I)) = Max (P(l, J)) 

J 

where 1 < J£ M, J assignable to 1 in the current state 
of the algorithm (see comment in Appendix 1). 

NBEST(I) is such that 

P(I, N BEST (I)) = Max (P(l, J)) 

J 

where l£J<M, J assignable to I, J * BEST(I) 

DEP(l) is the set of all objects such that, should an 
assignment of label to R(I) occur, then for 
II € DEP(I), P(I1,J) should be recomputed. 

EFF(I) is the set such that if II € EFF(I) is assigned 
meaning, then P(I, J) should be recomputed. That 
is, going through the classification tree to compute 
P(l, J) for various J r s, a question about the label 
of II was asked. 

Note that 12 e EFF(Il) s II € DEP(I2) . 
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ASSIGN (LEVEL.) contains the name of the regions 
assigned meaning in that level. 


Procedures 


INIT(I): creates P(I, J), EFF(I) and puts I in DEF(Il) 
when necessary. This procedure makes use, if 
applicable, of already assigned meanings to other 
objects in previous iterations of the algorithm, 
when running R(I) through the sequential classifica- 
tion tree. 

SORT: maintains structure allowing access to the 
object with highest confidence factor among the 
objects not integrated yet (see Ref. 8, 
pages 150-182). 


Algorithm Steps 

0. INTER = 4> 

DEP(I) * EFF(I) -- <)> I = 1, ■ • S N 

1. Call INIT (I) I = 1, • • •, N 

2. SEEN = 0 

LEVEL = 0 

VALUE( LEVEL) * 1 

3. Compute for I = 1, ■ * 7 8 9 , N 


CONFIDENCE (I) = If P(I, NBEST(l)) * 0 

— — - 


(if regions are allowed to be lf unidentified objects, 1 ' 
then °o should be replaced by special formulas). 


4. Sort R(I) by CONFIDENCE(I) 

5. Choose I 1^I<N where R(I) has no assigned label 
yet such that 

CONFIDENCE (I) = MAX(CONFIDENCE(K)) 

K 

where 1<KSN, R(K) i INTER 

6* Mark BEST (I ) as unassignable for 1 in the cur- 
rent level and for deeper levels. (This will make 
BEST(I) not assignable again to R(I) should the 
program get back to that point). Put I in INTER. 
Set LABEL(I) = BEST(I). 

7. VALUE(LEVEL + 1) + VALUE {LEV EL)* 

P(I p BEST(I)) 

LEVEL = LEVEL + 1; ASSIGN (LEV EL) = I 

8. If VALUE (LEVEL) < SEEN then go to Bl. 

9. If all objects are assigned meaning then: 

(a) SEEN = VALUE(LEVEL). Store current global 
interpretation as best seen so far, and (b) go to Bl, 


10. Update P(I1, Jl) for all II e (DEP(I) - INTERP) 
by calling ENIT(Il) to all those objects. 

11. For all objects for which P(I1, Jl) was changed, 
store old values with a level so that when undoing 
the recent assignment, the old values may be 
restored- 

12. Compute BEST{1), NBEST(I) for all I t INTER 
if there is an uninterpreted object 11 such that 
VALUE (LEVEL) * P(I1, BEST(Il)) < SEEN then 
go to Bl. 

13. Go to 4. 

Bl. If LEVEL = 0, then exit program. 

B2. Restore all old values which were changed in cur- 
rent level and mark all pairs of (I, J) marked as 
unassignable in that level as assignable. Undo the 
assignment in level, remove ASSIGN (LEVEL) 
from INTER. 

B3 ■ LEVEL = LEVEL - 1. 

B4. Go to 12, 
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(a) Original picture 


(b) Output of the nonsemantic region grower 


(c) Result of grouping regions by their assigned 
meaning, taking only regions which were assigned 
meaning with confidence over 10 to be mergeable. 


(d) Grouping regions by their assigned meaning, 
all regions considered mergeable. 


Fig. 1. Example of steps of analysis of a road scene m L . 

Jhis page is reproduced at the 
back of the report by a different 
reproduction method to provide 
better detail. 
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(a) Original picture 


(b) Output of the nonsemantic region grower 



(c) Output of the semantic region grower, pass 1 (d) Grouping regions by meaning with confidence 10 





Fig, 2. Another example of road scene analysis 
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(a) Left ventricular angiogram. Output (k) Early iteration of semantic region 

of the nonsemantic region grower. The grower 

stopping criterion is to stop when the 
merger gets down to ZOO regions. 



(c) Result of additional iterations ( d ) Result of still more semantic region 

grower iterations. The region grower used 
is the grouping of all adjacent regions 
which are assigned the same meaning by 
the sequential assignment procedure, 
before the first as signment with low confi- 
dence level occurs. On each iteration, 
the confidence threshold is lowered. 



This page is reproduced at the 
back of the report by a different 
reproduction method to provide 
better detail. 


(e) Final output. The heart interior is the 
dark center. Around it is the chest cavity 
and on the two sides there is the dark 
frame border. 


Fig. 3. An example of steps of analysis of an X-ray image of a left ventricular angiogram 
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